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Abstract 

We investigate the relationship between learning styles, online content usage and exam performance 
in an undergraduate introductory Computer Information Systems class comprised of both online video 
tutorials and in-person classes. Our findings suggest that, across students, (1) traditional learning 
style classification methodologies do not predict behavioral measures of online learning, and (2) 
working on the online content specifically during allotted class time is positively related to exam 
performance. Controlling for differences across students, we find (3) accessing content on non-class 
days (consistency) is positively related to exam performance, while (4) working substantially ahead of 
the scheduled content pace is negatively related to exam performance. 
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1. INTRODUCTION 

The use of online learning systems in higher 
education and beyond has dramatically 
increased in recent years (Azarnush et al., 
2013). Due to their web-based nature, online 
learning systems allow for the automatic 
collection of usage data. This, in turn, offers 
researchers and educators new opportunities to 
understand and improve student learning. 

Recent technological and methodological 
advances present an array of techniques that 
hold the potential to gain deep insights from 
these data. These advances have led to novel 
findings in such disparate fields as medicine, 
marketing, logistics, and education. The field of 


education is particularly concerned with 
developing methods for exploring the unique 
types of data that come from educational 
settings, and using those methods to better 
understand students and the settings in which 
they learn (Baker & Yacef, 2009). The present 
work aims to bridge nascent literatures, which 
use such data to predict individual learning 
styles as well as learning outcomes. 

Little is currently known about systematic 
differences across students in online content 
usage, and if these differences are associated 
with particular learning styles and ultimately 
with exam performance. In particular, one 
question yet to be addressed is whether 
traditionally-measured learning styles are useful 
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for predicting online content usage. Early work 
establishing conceptual models used to define 
learning styles dates to the mid-1980s, 
suggesting such conceptualizations may not 
apply as readily to today's learning environment. 

Specifically, the present work aims to investigate 
the relationship between learning styles and 
online content usage and the relationship 
between online content usage and exam 
performance. The work thus has theoretical 
implications for learner classification 
methodologies and practical implications for 
educators in mixed-format classroom settings. 

2. LITERATURE REVIEW 

Kolb's (1984) seminal treatise on defining 
student learning styles gave rise to a literature 
in the educational field implementing the "Kolb 
Model" to characterize students and assess their 
responsiveness to varied educational methods. 
Since then, new models of learning styles have 
come into practice, building on the work of Kolb 
(1984). Recent years have seen an explosion of 
data related to student use and interaction with 
learning content via computerized delivery 
methods, and a corresponding emergence of 
literature which applies conceptual learning style 
models to these new sources of information 
about learners (Khan et al. 2009, among 
others). The proposed work seeks to add to 
nascent literature which bridges conceptual 
models of learning styles (and their relation to 
learning outcomes) with the wealth of data now 
available to educators and researchers via new 
delivery platforms. 

The work relates to two distinct strands of 
literature. The first uses data from web-learning 
platforms to predict student performance. 
Baugher et al. (2003) were among the earliest 
to study this relationship, examining whether 
total hits or inter-class consistency of hits to a 
course content web site have any value for 
predicting student performance in a course 
supplemented with online activities, finding 
stronger effects for the latter. Grabe et al. 
(2005) followed with a similar investigation, 
finding positive effects of the availability of 
online materials on both class absence and 
overall grades. In contrast to these early 
studies, Abdous et al. (2012) found little 
relationship between online activity and 
performance. Romero et al. (2013) use and 
compare the results of an array of data mining 
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methods to predict student grades based on 
online usage data. 

Studies comprising a second and more recent 
literature use advanced analytical techniques 
and detailed web usage information to predict 
student learning styles. Lu et al. (2007) and 
Hung and Zhang (2008) are among the first of 
these, investigating the relationship between 
learning style measures, online behavior and 
learning outcomes. More recent studies on the 
same topic include Ballenger and Garvis (2009), 
Hung and Crooks (2009) and Bousbia (2010). 
Clewley et al. (2011) and Azarnoush et al. 
(2013) use related methods to detect learner 
styles, and examine how content delivery 
systems can be adapted dynamically to adjust to 
individual-user learning types. Recent reviews 
which more comprehensively detail these and 
related studies can be found in Romero and 
Ventura (2007), Baker and Yacef (2009) and, 
specifically for business-related disciplines, 
Arbaugh et al. (2009). 

Overall, existing studies yield some common 
conclusions. First, it is important to be aware of, 
and also detect, student learning styles— 
especially when using online courses, and that 
consideration should be given to the diversity of 
learning styles when designing and developing 
online learning modules (in terms of content 
presentation and design features, for example). 
Second, educational data mining combined with 
traditional statistical analysis can give a deeper 
understanding of the determinant of student 
learning and performance. The current study 
seeks to build on the work of Lu et al. (2007) 
and Hung and Zhang (2008) in that we apply 
data mining techniques to investigate the 
relationship between learning styles (as 
measured by a separately-administered survey) 
and online content usage, and then assess the 
relationships between online content usage and 
exam performance. 

3. METHODOLOGY 

We use a combination of server log files, exam 
scores, and surveys which were collected over 
the course of the Spring 2014 semester in three 
sections of CIS 101 (Introduction to Information 
Systems) at a mid-sized private university in the 
northeastern United States. CIS 101 introduces 
students to various aspects of developing and 
managing computer information systems and is 
a required class for all business freshmen. As 
part of CIS 101, students received three weeks 
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of intensive Microsoft Excel training online 
(referred to as the Excel Boot Camp). The Excel 
Boot Camp content is delivered online and 
consists of 22 lessons, each of which consists of 
a short video tutorial and an exercise. Although 
the Excel Boot Camp is delivered online, 
students were still required to attend class 
(Mondays, Wednesdays, and Fridays for 50 
minutes each). In each class, students were 
asked to work on the Excel Boot Camp 
individually. Although the Excel Boot Camp is 
self-paced, students were required to submit 
their completed exercises according to a pre¬ 
defined schedule (averaging about seven 
exercises per week). 

The web-based nature of the platform allowed 
us to capture user interaction with the content 
as recorded in server log files. The log files 
contain information about each user's online 
content usage, such as login and logout times, 
as well as the time spent on each page. 
Consequently, the log files provide a rich source 
to quantify various aspects of student online 
content usage. Specifically, the following 
behavioral measures were calculated in order to 
quantify online content usage: 

• time online during class time (hours 
spent viewing online content while being 
in class), 

• consistency (number of non-class days 
during which a student visited the online 
content before the beginning of the 
exam study period), 

• time online in exam study period (hours 
spent viewing online content during the 
exam study period), and 

• time online working ahead (hours spent 
working ahead of the class before the 
exam study period). 

Two weeks after the end of the three-week Excel 
Boot Camp, students were tested on their 
knowledge of Microsoft Excel. The exam 
consisted of 20 multiple-choice questions, most 
of which require students to download an Excel 
worksheet and perform analyses in order to 
derive an answer. All exam questions were 
directly linked to one of the 22 lessons in the 
Excel Boot Camp. Given the two-week lag 
between the completion of the Excel Boot Camp 
and the exam, students were encouraged to go 
back and review lessons in the Excel Boot Camp 
during the exam study period. 

At the end of the semester, students were asked 
to complete a survey on learning styles. The 
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survey was comprised of questions which form 
the basis for the Index of Learning Styles (Felder 
& Silverman, 1988; Felder & Spurlin, 2005) and 
the Kolb Learning Style Inventory (Smith & Kolb, 
1986). 

The Index of Learning Styles (hereafter "ILS") 
assesses learning preferences on four 
dimensions. Each of the four scales consists of 
11 items. For each item, students complete a 
sentence by choosing one of two options 
representing opposite ends of the dimension. 
The four dimensions are (see Felder & Spurlin, 
2005): 

• sensing (concrete, practical, oriented 
toward facts and procedures) or intuitive 
(conceptual, innovative, oriented toward 
theories and underlying meanings), 

• visual (prefer visual representations of 
presented material, such as pictures, 
diagrams, and flow charts) or verbal 
(prefer written and spoken 
explanations), 

• active (learn by trying things out, enjoy 
working in groups) or reflective (learn by 
thinking things through, prefer working 
alone or with one or two familiar 
partners), and 

• sequential (linear thinking process, learn 
in incremental steps) or global (holistic 
thinking process, learn in large leaps). 

The Kolb Learning Style Inventory (hereafter 
"Kolb LSI") assesses students' preference for 
perceiving and processing information. It is 
based on Kolb's experiential learning theory 
(Kolb, 1985), which posits that how a person 
perceives information can be classified as 
concrete experience or abstract 
conceptualization, and how a person processes 
information can be classified as active 
experimentation or reflective observation 
(Simpson & Du, 2004). The Kolb LSI asks 
students to rank order four endings for 12 
sentences according to how well they think each 
one fits them. Each of the four endings 
represents one of the four dimensions in Kolb's 
experiential learning theory, which can be 
described as (see Lu et al., 2007): 

• concrete experience (tends towards peer 
orientation and benefits most from 
discussion with fellow learners), 

• abstract conceptualization (tends to be 
oriented more towards symbols and 
learns best in authority-directed, 
impersonal learning situations, which 
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emphasized theory and systematic 
analysis), 

• active experimentation (tends to be an 
active, "doing" orientation to learning 
that relies heavily on experimentation 
and learns best while engaging in 
projects), and 

• reflective observation (tends to rely 
heavily on careful observation in making 
judgments). 

4. RESULTS 

A total of 91 students were enrolled in three 
sections of CIS 101. All students completed the 
Excel Boot Camp and the accompanying exam. 
Of those, 82 (90%) completed the end-of- 
semester survey on learning styles. As seen in 
summary statistics presented in Table 1 (see 
Appendix), students spent far more time online 
during class time (M = 6.36, SD = 2.42) as they 
spent online working ahead (M = 1.10, SD = 
0.79). On average, students visited the online 
content on eight of the 15 non-class days before 
the beginning of the exam study period. During 
the exam study period, students spent on 
average less than half an hour accessing the 
online content (M = 0.28, SD = 0.45). 

The correlations between all measures used in 
this study are presented in Table 2 (see 
Appendix). With regards to learning styles, we 
found only small to moderate correlations 
between the ILS and the Kolb LSI (all r < 0.32), 
suggesting that the two instruments measure 
different aspects of learning styles. The largest 
correlation is between the active-reflective 
dimension of the ILS and the active 
experimentation dimension of the Kolb LSI. This 
is not surprising, given that both of these 
dimensions measure a preference for "learning 
by doing". 

There is also substantial variation in the learning 
styles across students in the class, suggesting 
that the class composition was not overly 
skewed in terms of attracting only a certain type 
(or certain types) of students, as show in Figure 
1 (see Appendix). 

In order to examine the relationship between 
learning styles and online content usage, we 
conducted ordinary least squares regressions of 
learning style dimensions (both ILS and Kolb 
LSI) on the various measures of online content 
usage. Although there is substantial variation in 
the distribution of the types of students taking 
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the class (Table 1 and Figure 1), the regression 
results in Table 3 (see Appendix) show that 
neither of the learning style typologies yields 
strong predictions of online content usage. 
Among the learning style measures captured by 
our survey, only one type has a statistically 
significant relationship with any of the online 
content usage measures: a higher score on the 
"reflective observation" component of the Kolb 
LSI is associated with a higher likelihood of 
working ahead of schedule (although the effect 
is small: (3 = .035, p < .05). This finding 
suggests that students who are more reflective 
learners, and thus tend to rely more on careful 
observation, may be more intrigued to work 
ahead and explore the content ahead of class 
than students who are less reflective learners. 

We next investigate the relationship between 
online content usage and exam performance at 
two levels of aggregation: between students 
(i.e. across students) and within students (i.e. 
across lessons controlling for student-level 
differences). The latter level of aggregation is 
possible due to the server log data identifying 
which specific lessons a user was viewing, and 
for how long, and by linking exam questions to 
specific lessons we are able to construct a topic 
level panel. The estimating equation for the 
between-students regression is the following: 

(1) Score,- = /?o + Pi * class time spent online t + 

(3 2 * time spent online during exam study f + 

/? 3 * consistency t + /? 4 * 

time spent working ahead t + u t 

where /' indexes students, and all regressors are 
initially measured as described above. 

The results of the between-students regression 
analysis are presented in Table 4 (see 
Appendix). Across students, time spent online 
during class time (as opposed to self-study 
outside of class, as measured in various ways) is 
the single best predictor of exam performance (/? 
= .72, p < .01). In other words, for every hour 
(which is slightly longer than one class period) 
that students accessed the online content during 
class time, their exam performance increased by 
0.72 points (out of 20, equivalent to 3.6 
percentage points). This suggests that working 
on online content during class time is more 
effective than working on online content outside 
of class time. Given that this measure in essence 
captures class attendance, this finding suggests 
that irrespective or student-level differences, the 
single most important factor influencing exam 
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performance is coming to class and working on 
the assigned online content. Although this 
finding might not seem particularly novel at first, 
we believe this points to the importance of 
blended learning strategies that combine online 
and in-person classes. 

The results of the between-students regression 
analysis paint a different, more nuanced picture. 
The estimating equation is: 

(2) Score, j = Po + Pi * class time spent online^ + 
P 2 * time spent online during exam study f j + 

P 3 * consistency^j + p 4 * 

time spent working ahead,j + I’ * X, + u, y 

where / still indexes students, j indexes lessons, 
and Xy is a vector of student fixed effects. 

Table 5 (see Appendix) contains the results of 
the estimation of equation (2). When accounting 
for individual differences, time spent online 
during class time is no longer a significant 
predictor for exam performance (/? = .0004, p > 
.05). In contrast, we found that consistency, as 
measured by the number of non-class days 
during which a student used the online content, 
is significantly related to exam performance (/? = 
.073, p < .001). In other words, for every non¬ 
class day that students accessed the online 
content for a particular lesson, their chance of 
correctly answering the exam question relating 
to that lesson increased by 7.3 percentage 
points. Interestingly, the significant negative 
coefficient for time spent online working ahead 
of the class (/? = -0.11, p < .01) suggests that 
working ahead decreases students' exam 
performance. Specifically, this suggests that for 
every hour spent working ahead of the class, 
students' chance of answering the exam 
question relating to the lesson that they should 
be working on decreased by 11 percentage 
points. 

5. CONCLUSIONS 

The aim of this study was to understand the 
relationships between learning styles, online 
content usage, and exam performance. We 
analyzed a unique dataset that gave us rich 
information on student access patterns of online 
content that was part of a hybrid university 
course. Our findings suggest that (1) traditional 
typologies of learning styles may not accurately 
classify the different ways students have of 
interacting with online content, (2) the number 
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of hours spent working on the online content 
during class time is positively related to exam 
performance, (3) the number of non-class days 
during which students access online content is 
positively related to exam performance, and (4) 
the number of hours that students work ahead 
of the class is negatively related to exam 
performance. Based on these empirical findings, 
one can deduce three prescriptive guidelines for 
instructors using online content in their classes: 
(1) ensure that students work on the online 
content during class, (2) encourage students to 
review online content between classes, (3) 
discourage students from working ahead of the 
class. These suggestions should help students 
make the best use of the online content and 
improve exam performance when delivering 
content in a hybrid format. 

There are a few caveats to the analysis. 
Specifically, a relatively small sample from a 
private university in the Northeastern US is 
probably not representative of the general 
undergraduate student population. Also, the 
hybrid class format of combining video-based 
tutorials with in-person class meetings is unique, 
which might limit the applicability of our findings 
to traditional in-person or purely online classes. 
Lastly, we collected the survey of learning styles 
at the end of the semester. It is possible, though 
we believe unlikely, that the experience of the 
Excel Boot Camp had an effect on students' 
responses to the survey. 

This study adds to an emerging literature using 
large datasets that capture detailed information 
on student interaction with educational content 
to detect patterns of content usage and predict 
learning outcomes. There are several directions 
in which future work in this area can go. The 
first is to use such data collection for semester- 
long classes, which will provide richer variation 
to use in classifying student interaction with 
learning materials. Another will build on this 
type of data to generate new learning style 
typologies that are more suited to online 
learning behaviors. 
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Appendix: Tables and Figures 


Table 1: Summary statistics of online content usage, learning styles and exam performance 



mean 

sd 

median 

min 

max 

N 

by student: 

time online during classtime (hours) 

6.36 

2.42 

5.89 

2.07 

13.35 

91 

consistency: non-class days visiting content 

8.02 

2.58 

8 

1 

15 

91 

time online in exam study period (hours) 

0.28 

0.45 

0.00 

0.00 

2.06 

91 

time online working ahead (hours) 

1.10 

0.79 

0.85 

0.13 

3.65 

91 

Active-Reflective (ILS) 

-1.37 

3.98 

-1 

-9 

9 

82 

Sensing-Intuitive (ILS) 

-2.35 

5.64 

-3 

-11 

11 

82 

Visual-Verbal (ILS) 

-4.17 

4.65 

-5 

-11 

9 

82 

Sequential-Global (ILS) 

-1.78 

3.51 

-3 

-9 

9 

82 

Abstract Conceptualization (Kolb) 

31.00 

5.73 

32 

15 

44 

82 

Active Experimentation (Kolb) 

31.85 

8.40 

33 

16 

48 

82 

Concrete Experience (Kolb) 

28.77 

7.89 

29 

13 

46 

82 

Reflective Observation (Kolb) 

28.28 

6.94 

27 

14 

48 

82 

Exam score (# correct out of 20) 

16.18 

3.40 

17 

5 

20 

91 

by student-lesson: 

time viewing lesson content during classtime (hours) 

0.32 

0.30 

0.26 

0.00 

2.04 

819 

consistency: non-class days visiting lesson content 

1.04 

0.71 

1 

0 

4 

819 

time viewing lesson content in exam study period (hours) 

0.02 

0.06 

0.00 

0.00 

0.54 

819 

time viewing lesson content working ahead (hours) 

0.11 

0.24 

0.00 

0.00 

1.96 

819 

% correct of questions asked on lesson content 

0.79 

0.35 

1 

0 

1 

819 

across individual pageviews: 

time on page (individual view; in seconds) 

762.99 

769.21 

569 

60 

7190 

2829 


Notes: Table presents summary statistics of online content use, learning styles and exam performance at 
various levels of aggregation. 


Table 2: Pairwise correlations of online content usage, learning styles and exam 

performance 



1 

2 

3 

4 

5 6 

7 8 

9 10 

11 12 

1. Exam score 

1 








2. time online during classtime 
(hours) 

0.26 

1 







3. time online in exam study 
period (hours) 

0.17 

0.16 

1 






4. consistency: non-class days 
visiting content 

0 

-0.3 

0.2 

1 





5. time online working ahead 
(hours) 

0.05 

0.01 

0.09 

0.21 

1 




6. Active-Reflective (ILS) 

-0.03 

-0.05 

-0.07 

-0.07 

-0.15 1 




7. Sensing-Intuitive (ILS) 

-0.1 

-0.12 

-0.01 

-0.03 

0.23 -0.29 

1 



8. Visual-Verbal (ILS) 

0.06 

0.05 

0.21 

0.15 

0.02 0.14 

-0.23 1 



9. Sequential-Global (ILS) 

-0.03 

-0.2 

0.05 

0.05 

0.2 -0.01 

0.21 -0.05 

1 


10. Abstract Conceptualization 
(Kolb) 

-0.05 

-0.06 

-0.04 

-0.08 

0.01 -0.06 

-0.18 0.16 

-0.16 1 


11. Active Experimentation 
(Kolb) 

-0.15 

0.06 

-0.09 

-0.12 

-0.09 0.32 

0.04 -0.01 

-0.1 -0.14 

1 

12. Concrete Experience (Kolb) 

0.09 

0.01 

-0.01 

0.06 

-0.12 -0.04 

-0.17 0.03 

0.11 -0.39 

-0.53 1 

13. Reflective Observation 

0.14 

-0.02 

0.16 

0.13 

0.24 -0.28 

0.3 -0.18 

0.15 -0.22 

-0.5 -0.17 


(Kolb) _ 

Notes: Table presents pairwise Pearson correlation coefficients across combinations of student-level measures. 
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Table 3: Predicting online content usage with learning styles, OLS regression results, 


Dependent variable 

student-level variation 

time online time online in 

during exam study 

classtime period (hours) 

(hours) 

consistency: 
non-class days 
visiting content 

time online 
working ahead 
(hours) 

Panel A: Using ILS 

Active-Reflective (ILS) 

-0.024 

-0.011 

-0.064 

-0.022 


(0.04) 

(0.01) 

(0.07) 

(0.02) 

Sensing-Intuitive (ILS) 

-0.022 

0.0001 

-0.015 

0.026 


(0.03) 

(0.01) 

(0.05) 

(0.02) 

Visual-Verbal (ILS) 

0.007 

0.020* 

0.084 

0.014 


(0.03) 

(0.01) 

(0.06) 

(0.02) 

Sequential-Global (ILS) 

-0.06 

0.007 

0.045 

0.037 


(0.04) 

(0.01) 

(0.08) 

(0.03) 

Constant 

1.621*** 

0.350*** 

8.541*** 

1.275*** 


(0.21) 

(0.07) 

(0.44) 

(0.14) 

Observations 

82 

82 

82 

82 

R2 

0.052 

0.057 

0.036 

0.094 

Panel B: Using Kolb Learning Styles 

Abstract Conceptualization (Kolb) -0.012 

-0.001 

-0.036 

0.012 


(0.03) 

(0.01) 

(0.05) 

(0.02) 

Active Experimentation (Kolb) 

0.006 

-0.001 

-0.028 

0.007 


(0.02) 

(0.01) 

(0.04) 

(0.01) 

Reflective Observation (Kolb) 

-0.003 

0.009 

0.021 

0.035** 


(0.02) 

(0.01) 

(0.05) 

(0.02) 

Constant 

2.059 

0.084 

9.646*** 

-0.484 


(1.63) 

(0.55) 

(3.25) 

(1.03) 

Observations 

81 

82 

82 

82 

R2 

0.006 

0.026 

0.026 

0.068 


Notes: Table presents coefficients from a linear regression estimating various aspects of online behavior. 
Standard errors in parentheses. Significance levels indicated by * .10, ** .05, ***.01. 


Table 4: Predicting exam scores with online content usage, OLS regression results, student- 


level variation 

(i) 

(2) 

(3) 

(4) 

time online during classtime (hours) 

0.720** 

0.450** 

0.616** 

0.412** 


(0.31) 

(0.18) 

(0.31) 

(0.18) 

time online in exam study period (hours) 

0.846 

0.11 

0.869 

0.104 


(0.82) 

(0.10) 

(0.80) 

(0.10) 

consistency: non-class days visiting content 

0.062 

0.067 

-0.051 

-0.039 


(0.15) 

(0.16) 

(0.15) 

(0.16) 

time online working ahead (hours) 

0.131 

0.191 

0.099 

0.092 


(0.46) 

(0.52) 

(0.45) 

(0.52) 

Constant 

14.021*** 

10.055** 

15.252*** 

12.151*** 


(1.47) 

(4.48) 

(1.53) 

(4.55) 

Observations 

91 

91 

88 

88 

R2 

0.089 

0.089 

0.082 

0.087 


Notes: Dependent variable exam score. Table presents coefficients from a linear regression 
estimating exam scores with measures of online behavior. Columns 3 and 4 exclude a 
small number of students with outlier values in online use measures. Columns 2 and 4 
estimate the specification using natural logs of the online time measures. Standard errors 
in parentheses. Significance levels indicated by * .10, ** .05, ***.01. 
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Table 5: Predicting exam scores with online content usage, OLS regression results, student- 

lesson variation 

m m 


time online during classtime (hours) 

0.0004 

0.0003 

spent on lesson 

(0.10) 

(0.01) 

time online in exam study period (hours) 

0.004 

0.007 

spent on lesson 

(0.25) 

(0.01) 

consistency: non-class days visiting lesson 

0.073*** 

0.066** 


(0.02) 

(0.03) 

time online working ahead (hours) 

-0.110** 

-0.007* 

spent on lesson 

(0.05) 

(0.00) 

Student fixed effects 

Y 

Y 

Observations 

819 

819 

R2 

0.886 

0.886 


Notes: Dependent variable: Percentage correct of questions 
pertaining to lesson. Table presents coefficients from a linear 
regression estimating question scores across lessons and students. 
Column 2 estimates the specification using natural logs of the 
online time measures. Standard errors in parentheses. 

Significance levels indicated by * .10, ** .05, ***.01. 


Figure 1: Histogram of learning styles across students 
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